Distilling and Transferring Knowledge Via Cgan-Generated Samples for Image Classification and Regression

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a student based on knowledge from teacher. However, applying KD regression with scalar response variable rarely studied, and there exists no method applicable both yet. Moreover, existing methods often require practitioner carefully select or adjust teacher architectures, making these less flexible practice. To address above problems unified way, we propose comprehensive framework cGANs, termed cGAN-KD. Fundamentally different methods, cGAN-KD distills transfers model via cGAN-generated samples. This novel mechanism makes suitable tasks, compatible other insensitive architectures. An error bound trained is derived this work, providing theory why effective as well guiding practical implementation Extensive experiments CIFAR-100 ImageNet-100 show that can combine state art yield new art. Steering Angle UTKFace demonstrate effectiveness where are inapplicable.